ee eS 


The views expressed in this paper are those of the author 
and do not necessarily reflect the views of the 
Department of Defence or any of its agencies. This 
document may not be released for open publication until 
it has beon cleared by the appropriate military service or 
government agency. 


ome 


iy 
BEN ye Rh ee ok Per pis 
C { vege tie ee eee GES bei 


meeiine 53: Shan pepe ro preety 
PER fis d Pb: id j bo rock PRCGSG Le vi 


IMPLEMENTATION OF RISK ASSESSMENT IN THE 
TOTAL RISK ASSESSING COST ESTIMATE (TRACE) 


DA041467 


A LIEUTENANT GOLONEL. GENE A Venzxa> 
: IGNAL CoRPS 
OW; : are ¥ a 
oO 4 COPY AYALATE | [lo Pons 2 
! Jud 4 PER IT FUL L i UGEr: Y me we) Ah 
fous A ial 


WS ARMY WAR COLLEGE, oe boulcchetind a8 ue 


k Ay PRE RMOR for public relense: 


+ sa ereneperemmnane 
Meine cas: Mtie 


Retell eek A 


Best Available Copy 


Me 


ete ere Bo rE ste atte ered ides ooo skein mace emo 


4 The views expressed in this paper are those of the author and do not 
necessarily reflect the views of the Department of Defense or any of its 

agencies. This vocument may not be released for o,en ule: aus unit it 
hes been cleared ty the appropriate military service uf government agency. 
ee eames nie 


CANEPA RTT reap geipce a Week, Gee cred 
TRESS Toe ANSE ASB a Se TT BES SPA PAP: 
—" 
Sat Garcia oes 
ee eh : a asl 
Ae tire Seer EO, EV re) Ah ee ee a eit 
- my 
re eatte Riepe ene ee ies te 
Wy re oe A reek ro Pormtry en eae | re Oe 0 Oe | 
: Rb area 


fo ath SS OWE yt ~ eo 5 
: : yea imp erye HEY CE tees 
wee, Ree aie pea ee ae SH Chee 


oe 
Oy 


Lice ereth, SoLsee) Tene vs. TOURS, Stee 


Colones. Rovers |. Jead, AR 


Otay &avicor 


Rae shies, Soothe Me rts 
oe Saar Sa dap Goll 

wt ae sre <e 

Cerplicole “arracks, Ferns, 


’ 
25 say 1977 


gu 
Mae 


a a I IT OI TD, 
Approved for public release; 


distribution unlimited. 
ee aaEanemnanrsammenemmenamnen 


te hastaienl 


othe Deper stonse cy eny cf Its 
Legit Open: GUDLG wn unl it 
peptione 'waltiery :<ervice or government sgency. 


USANS ee STUDIES ) PROGRAM PAPER 


. OF FOR HO % 
’ 


baba oa mtlia , 
j dean: b 


‘ .¥ oo ud gen par 


Leet Saeerent® & 


TMEESr EN - arn ae : ssa att 
HRiete SONY) ean OF OF | RISK ASSEZSSMENL? IN THE 


heen ths siti es “ 
“OTAT. RISE ASSESSING COST ESTIMaArRe (TRACE ) 


AS INDIVIDUAL STUDY PROJECT 


Colonel Robert f. Reed, AR 


“tudy Advisor 


vo Army Var Collese 
“erliscle arracks, Fennsylvania 17013 


25 vay 1977 


Cs Approved for public release: 


distribution unlimited. 


» REPART NUMBER 2. GOVT ACCESSION NOJ 3. SEMIFICN I'S CALALUY nUumoEn 
(2 = ae eae 

MPLEMENTATION OF RISK ASSESSMENT IN THE JOTAL Heed) Study Project 

| | frst, ASSESSING Cost t ESTIMATE (TRACE), ~ a 


RFORMING ORG. RE PORT NUMBER : oe 
Fe. 


+ PERFORMING ORGANIZATION NAME AND ADDRESS ats ROJECT, Task 


6. CONTRACT OR GRANT NUMBER(c) 


a 
US ARMY WAR COLLEGE 
Carlisle Barracks, PA 17013 


crite Lecabeabasbt sit dist sia coidet 6 ie 


Pon 


Fat ee? WH 


411. CONTROLLING OFFICE NAME AND ADDRESS REDOA 3 
Same as Item 9. (/[) | 28 Mey B77 4 
j 
4 MONITORING AGENCY NAME & ADDRESS(IT Gane from Controlling Office) | 18. SECURITY CLASS. (of thie report) ; 
by Unclassified i 
; 98p o 
2 15a. DECLASSIFICATION/ DOWNGRADING j 
a SCHEDULE H 
a 16. DISTRIBUTION STATEMENT (of this Report) : 
ee e 
‘7 Approved by public release; distribution unlimited. : 
‘ j 
we 2 
: OD eM 
. c) x : 
a — es + 
F 17. DISTRIBUTION STATEMENT (of the abetract entered in Block 20, ff different from Report) oe, \] 
& ee Co ‘ “oN 
| " \- aye 
b \ ars: 
Q — —, 
i '@ SUPPLEMENTARY NOTES Woe 
H ‘ 
i 
i ' 
cm iS 7 
t 19. KEY WORDS (Continue on reverse elde if necessary and identity by block number) : 
: 
er ee eae oe a 
é boo. y ac ae i ee H 
i * eee eee F by powe wee & ae ee we : 


20. ABSTR (Continue on reveree side if mroeneeT anc Identity by block k number) 


t She concept of the otal tick Ascessing Cost Estimate 
“C(EYACT vias achiculated a tie ASA(R%D) on 12 July 1974. It is 
‘9 jagans of explicilly oumecemodating tne unforseen and un- 
Hlentifiabls co: ls viien eiaructeorize research and development 
pradeate. Mae VPC S i reagnired to possess the property tnat it aa 
i an catdmale of {yo SObh nercentile of the project cost = es 
wpobalility dicheibution, jinfortunately, carly attempts to “-~ 
| A j 


[of] fie ae 1473s EDITION OF | MOV 65 IS CRSOLETE 


SO4 5S o" SECURITY SURITY CLASSIFICATION OF THIS PAGE (When Dete Ent 


FASS i EERE I I teen ee aE Pa ER, LS ONE | oa ee AER 2 A ene oe 


BRR me 


leploson! Lae ORAS aet vith limited success. A formalized 
study vac uniertakan to develop adequate techniques and two 


candldate petnodalosins emerged. One of the approaches, the eet 
SANT Malyork "adel, ic othrenely promising. The second tech- 
wiqus, TMG? Fink abulalton, ean be improved upon by a mod- ° 


Fleation tnvalving corputer generation of the imbedded 
proweility distribution. There remain some problems in “edu- 
epting” users af the value of the TRACE, and the TRACE concept 
saefers Cron neae luasrent shortcomings. It 16 recommended that 
the naw heeiniques for developing the TRACE be implemented, 
Along with some aucillary actions to support the implementation 
and onbance the usefulness of tue TRACS.. 
ANS 


NUS ORs 


A 


Tene A 


qv Q 
FONT 


wes imate. ft 
ote 


Slay wa undorhelan (ta aeye 
: 
candidate ipetiedatosies omeap 


: 4 ; 6% : ea 1 ° i, i 


yey 


"TPES: Taplementation of isk A 
Assessing Cost &stimate 


FORMAT: Individual Study Project : 
DA'TE: 20 way 19/7 FAGES: 52 CLASSIFICATION: Unclassified 


he concepL of the total aick Assessing Cost Estimate 
PER OEN, oye: Aritondlated ty the AcA(RD) on 12 July 1974. It is 
A ee OT “aol toll ly aesomnodatling the unforseen and un- 
flentiftabls eoclug waten eharucterize research and development 
“1 CUMS is peduired to possess the property that it 
SoA sot lame OF fie S0% uereaentile of the project cost 
sypobabalite 44etepha#dar,, ‘Infortunately, early attempts to 
ieetanen’ bl Mens met atta MMmtithed Success. A formalized’ 


oy 


fOd,. 


ON Otome MATAL 9 4s cvLesaoly promising. The second tech- 
uk ‘atsulat ton, ean be improved upon by a mod- 
VSteation iavolving corguler generation of the imbedded 
peoooidity disteibilion, There remain some problems in "edu- 
eg MOI OE Ase Sea ag ah Lice TUS, and the TRACE concept 

iSPors Spee awe Dnyerent Seolconmings. It is recommended that 
rep wen Peelmbaas: Por doveloain 

hip Mh wun anetliare aettons to Support the implementation 

LUNE ot Gece eS rea RWC cer eae an LON Ss. 


ssessment in the Total Bisk 
(SHA Cty) 


adequate techniques and two 
One of the approaches, the 


= the TRACE be implemented, 


ee ge Nee oe S 
Pas Gis 
os Bat dower t 
“2 
i ss 
cr, 


tt 
ie o 


Tok" As p 


-AVALASLE COPY 


PateD rte AC Ne a 


iis Individual Study Project was undertaken vs 
‘ 


. ) 
Ron Saadeamia Year 19997. “he study responds to a tople 
Siggestion Sw the Desuty Chief of otaff for Researcn 
neveloammont ant Soquicilion (Meo DA), Jeadquarter: 
tuo Aras. “ne author's enoies of this 
Oe le pad upon Sxperionce with the problem at tue 
Meojeet Vanagemet Level combined witn @ cont inuin ae 
in? orog) dy lanrnovine ine tools avallabiea co: ue nrg 
SNe ave sosant of “lessare..: and Developneny projects. 


ace 


eae eee Rees: See: Sys as study, tHe author wae 
Sietlecet bs secures tne cooneration of individuals fron 
TLE Ras ee ane soa OTP i oes. -(inelud lie 6. leo jecs 
ERAS INS rpoprdale Pune tiontl (Suaty ofl pads of “tapos 
toe Subordinate Jemmands of bhe Rea Oye Tain Lea 
el Ainesc Tneand fame) , mat well AS Numerous personne 
Pr, SAO, within the Crloids, ovner os Suer?: ions 
aT iM ya Awe Cenre Aplat,. he roader will find, “ouever, 
Ge ee ames to specle te: Ilividua ls, ae te Inforie. ron 
Sas eee vie Le Ov A onorea CUP thion bese ta sricoura:; 
PW. Sa Sane nt Ay So lOeupe. Very Bll oP u- tS Deen 
Ss rape One and halanee. progentacioa ef tae 
Eye These ek age: 39 30 Leer tg 
We SP he PEO: A AOR SPAS Bee OORT 1 Oc O. ats 
eee ei SOL. Ge eke splendid CeOp EEL 
ee, Geet oe ae Leangiand, Sigg elon ead bo wie guans RRS: 
Bre SR Se Mee iyoorsgecs Peni OF oer Sis OG. 7 ee 


ae aig tar nse fad Westen ‘Te Mage saad obey Rises 
J od7tocor, Ta) Teherl T reed, ame .to tne 
a oc 2 rr ok yal goatee sey nrawi Od, yered ny 


TRANS UOT PAN An OTE Ree ag 2 INO” ANNE Va 
. ee oS ce ‘ : 
vO Meow Pm omg trod suenars,. 
. - “a on; J Le at emt 


Best Available Cop 


a Away dar Collose Nilitary Studios frogran 


TABLE OF “CONTENTS. 
ABSTRACT Se ee er re Er are ee ee a ee ee ee ee 
GA PPT ER T * INTRODUCTION. eeeneeeeee” eo #eee eevee e te ee 
The Concent of the Total Risk 
Assessing Cost Estimate (TRACE) . 
Project Costs as Probabilistic 
Sntities eseeeooeveoeveoeve et Ooveeveeweeaeeee 
The TRACE in the Probabilistic 
MYAMEWOPK ccoccccscccsesessensvece 
TRAC'S: DOLINL ELON. ais si oWiete eee eo 
Statement of the Problem. ..ccccscons 
TT. CURRENT AND PREVIOUS TECHNIQUES «20.6 
axeursions from the Baseline Cost 
MSU IMAGCE: 444 4 sve sce Wea e bie ee ee Steere s 
‘istorically | Rased Exaursions.cecee 
es Rased "XCUrSLONS .eccvceee 


Indenendent raranebrie Cost ustimates 


“enaserent Neserve and elated 
TECKNIQUeS coccesccccceveeneseces 
TPP EVOLVING “CNT QUIS ee ceee ee eeeeeres 
TINCT Network Modeling-Deserl ption 
“TAC Network “odeline-Uiscussion : 
ee eek fabulation-deseription : 
VEN Ge eee tabulation- Discussion .. 
as creme Pee TG OL GUase a 
“4 ULM STON / ELPA NS TOM See ee ee 
Peak tan POrMULAGLON scececceecncnes 
“onrutation of the Prohability 
Yistribution Coen eeevreeersre ene eseerns 
‘ee of tne Probahility Distribution 
ALAGUuaAtOnoes Khe. Provored ASoroic 
ERE 2 EMO Gig ele oe 9/66) we Sw 
Pat eae SE TOUS C.- PAA 


daar Sage ae, Par epeyst 


eet ener ane eee eeers eons 
. an 

ys ar For Tsar ES , eoveeeesne ae see 
aha ans he “dt: a nF fae sat aN seve eoene 
77 = rreyynd, ah 4.07 ce a a eas eeoereervrovoese 
ay nagtin De ORD EAS eC OO RON. « 

. ghe ame : zi 
an : Po 3 “ oeveveereovreere eae seeens 

Ce ee 

) 
~ 2 nat ’ ay ae NE Wh eevee evr rere e eee ee 
= : : é » hea he . ooeaeeerpeasvseevee eee eeee 
SD ee oe ee Pen Re Mes eT og 
ot MPO ome e cee cee er enor e newer enneseesoenn 


3est Available Copy 


wa 7 EH + = cae at 


qe GA 
Latah tn 


~ 


Sra 
wd ferntnnd} 


~ 
-J 


NALAA 


i a ‘ad ones! Li) 


aon 


CHAPTER T 


ye aay 48 
‘8 


INTRODUCTION 


Prior to developing techniques for implementing 
Risk Assessment, it is necessary to introduce some 
general concepts and definitions which will serve as 
the basis for detailed discussion. This’ chapter 


provides that background and introduces the problem. 


CUE oe Qf THE TOTAL RISK : 


G COst ESTIMATE 


it is relatively common knowledge that a serious 

problem in defense development programs is cost over- 
runs and schedule slippase. Very few proyrams are 
eo mpleted witnin initial estimates of cost or within 
‘ne time period initially projected for tne development. 
“ost averruns of 50 or 100 per cent are not uncommon, 
1s ean veatily be determined by perusal of newspapers 
or news magazines on an almost random basis. Those over- 
runs, in addition to areabting adverse publicity and 
cobrermicaumant ,peeoont a substantive problem -- to 
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other projects of lesser priopity. The net result 4s 

that tne Army gets fewer of the systems on which plans 
were based, money spent on deleted systems is wasted, and 
deferral of other systems generally induces cost growth 
in those systems, thus creating a "snowball" effect. 

On 12 <tuly 1974, Mr. Norman Augustine, then 
Assistant Secretary of the Army for Research and 
Development, in a memorandum to the Director of the 
Army Staff (Annex A), directed that henceforth the con- 
cept of the TRACE would be used in Army Research and 
Development (R&D) Programs. The basic philosophy of the 
TRACE is as follows: Inherent in any R&D program iS a 


factor of risk. Since R&D implies a foray into the 


unknown, it cannot be expected that every activity will 
occur according to plan. There will be blind alleys, 
unforseen complications, and similar mishappenings. 


Ts is therefore unrealistic to allocate to a project 


only those resources wnich correspond exactly with the 
project nlan. Additional funds myst be programmed as 
A marzin arainst those unforseen and unidentifiable 
arohlems snien characterize R“D projects. 

“me TRACE concent can be readily displayed in 


“raonic form as shown in figure l. 
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BASELINE CCST 
” ZSTIMATE (BCE) 


INCLUDES ALL IDENTIFIABLE 
COSTS AND IS USED BY THE 

PM FOR DAY TO DAY PROGRAM 
MANAGEMENT 


FIGURE 1 
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It should be noted that the, Project Manager (2M) 


continues to use only the planned activities (Baseline 
Cost Estimate or BCH) as the project target cost, while 
the TRACE Margin is held at Department of the Army (DA) 
level in the Office of the Deputy Chief of Staff for 
Research, Nevelcpment and Acquisition (DCSRDA) for 
release to the Pil uvon application and justification. 
However, the TRACE value is used for programming and 
budget purposes, and the cost entry in the Five Year 
Nefense Program (FYDP) is the TRACE. Thus, the Army nas 
fewer, but more realistically funded projects in the FYDP. 
After TRACS funds are exhausted, should that occur, tne 
project may be continued through reprogramming, if 
aesired, but is subject to challenge by higher authority 
5O include action by the Defense Selected Acquisition 
Review Council (DSA8C), Should it appear that the 
NeciSion Coordinating Paper (DOP) tnreshold will be 


hraacned. 
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"Oo pur the TRACE concept into a broader perspective, 
Ye in useful to introduce a probatilistic view of project 
enube. ila whlew uses the notion that project costs 


Staaf nese li by attri bution inem as Ligure 2. 
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$200 = $400 $600 +800 $1000 
COST (in millions) 


Sigure 2 


Tho probability distribution is interpreted by deter- 


mining the area under the curve for any cost range of 
interest. ror example, the probability that the project 
will cost less than a given amount is the area under 
rae eurve between zero and the given amount. Figure 3 


tllustrates this for 3400 millim. 
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Sinee the shaded area in figure 3 is 0.35, it is 


determined that the probability of a project cost 
less than or equal to S400 million is 0.35. By 
convention, $400 million is called the 35th per- 
centile of the distribution. 

Similarly, the probability that a project will. ineur 
a cost greater than a given amount is found from the 
area under the curve to the right of the given amount. 


“irure 4 illustrates this for $600 millicn. 
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Mr, Augustine, in his 12 July 1974 memorandur, 
rie 


Perea 


Tr yh ore 


Cees whos, MASITGR ly. Ne stated: TA tne Peis sould: 
be develonea so as Lo represent the modiarn, or 50th 


percenlile of tha cost probanility distribution of 


a project. Figure 6 illustrates tnis concept. 
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wre” Mis there is a 50 ver cent chance tnat tae project. 


er wili cost lese than the TRACT (underrun) and likewise a 
50 per cent chunce taat it will cost more (overrun). Ur. 
Augustine did not nrovide a rationale for selecting tne 
pedian as the TRACE, but a reasonable rationale may be 
Seduced as follows; the basic goal of tue ‘TRACS 1s to 
minimize the requirement Lo cance] or defer on-going or 


planned projects in oreder to fund overr 15 in other 


spojects, by cnoosing the [RACs ab tine wedian cost 


for each projeat, we achieve a solution where in the 


i ’ 
lone run, about half tae projects will overrun, while 
nalf will underrun. Cveralil, the money 'released* by 
Lhe underruns Soul. caver the cost. or the: overruns, 
and the necessity to disturb the programed funds 
srould he avoided, 

Tf viered Jt tae Suort run, this rationale nas 
some defects. Most Significant is tnat in any given 
year, one may Tind tnat tne overruns are ell in tne 
lareer projects, waile tae underruns are in tne smaller 
projects, resultins in a net snortage of funds. Obviously, 
tne reverse covld just as easily occur, with a resultant 
excesa of funds on hand. If tne state of tue art of 
cost cstination permitted, ane could actually adjust 
the TRACE percentiles (i.c., make the THACH percentiles 
Ayfferont from the median) on a project by project 


nmasis So that there vould be a hien probability of 
offsetting underruns and overruns. However, the slate 

of the art of cost estimation does not supporl so 
ambitious an anproacn, and implementation of tines median 
approacn provides adequate challenge, as well as offering 


Simnificant, if not ultimate improvement in reducing 


resource programming turbulence. 
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Unfortunately, in day to day use at various offices, 


the ‘Padce has acquired a large number of definitions and 


interpretations. “or nurposes of insuring clarity and 


Pa) 


understanding in tnis paper, and as a proposal for an 


overall definition to improve communication throughout 


my 


the Aruy, tne followins definition of the Taace is 

given: 
"one Vlotal Risk Assessinm Cost wstimate (GRACE) is 

a point astinate, given in dollars, of tne cast of a 

research and development project. Tne estimate is 

chosen sucn tnat tne nrobaility that it will be exceeded 

is 0.50. "or control purposes tne TRACe is normally 

divided into two parts -- the naseline Cosl Estimate (Cs), 

whien includes both known and anticipated costs, and 

thie PRACH Marcin, wnien is a provision for the uaknown 

anil unidentified costs. The TAACE (and ibe components) 

may oddress the overall] project cost, or toe cosl for a 

riven “iseal Year. TP nol clear Troa context, tris 

distinction should be made explicit by identification, 

cuci as ‘the "YOO TRAC" to indicate only one. year, or 


‘Lhe orojecl TRADE! to indicale overall project cost." 
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The oroblem is hardly new. There has been a con- 


industry and in government to improv: tne accuracy and 
realicm of cost estimates over tyre years, and in fact, 
cose eshLimatines teenniques nave imrroved. Unfortunately, 
the availability of resources (in real terme) tas con- 
currently decreased, and thc ontions for conmillinnent of 
tnose resources have become more expensive. In snort, tne 
criticality of having &@ good cost estimate nas out- 

maced the improvement in cost estimating tecnniques. 

aus tne cnallenge is to provide a a@osh estimating: tu- 


provenenk wailed closes tic gap. This chapirr will diseuss 


Ps 


mposenes currently and previously ued, and assess 


tneir success or lack tncreof. 
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ef cost estinetes revolve around tne use of tuo HC! ag 
a point of departure and then developing an increment to 


eover polential cost srowbn. Wiis is besically an attenrt 


to develon tye TRAC Marin as an isolated ontity. wo 


wea hes 


basically different suhapproacies have been used. 


oe 


Askorically “Yased txevresion: 


Cre: Ie lemenbe lion of tives Leckinigqua wen. Bus ores 
by Vr. Ausuctines in his 12 Tuly 1974 incmorendum. Tt is 
connonly referred to ac bane "i" factor approawcno, altnougn 
the derisive term "Cudze factor" has also heen aprlied. 


there are many variations, but a cimple cxamnle will 


seyye to demonstrate the teennique. 
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rose the RC® for a new tank includes three majer 
fegorles; automotive, armament, and firc control.) «rom 
a roview of past tank developments, tne analyst d¢termines 


Liab Cost sPovtn in those threc catesorices aan: averared. 


oO. Ser eon. MOo nercent, and G0 ner-cenly ‘resuaccivel) . 
The current estimates in the three categories are then 
inersased. by tre corresponding cercentages. In 
of course, considerably more detailed procedures are 
used, to include a lover level of detail, wsc oF 

slatisliesl procedures to validate tne results, and 


inberrsbing scoodula considerutions jnalo the procedure 


ar 


to allow prosranming by Piscal veart 


These procedures suffer fron a number of defects. 


Pirst, tye use of the 3Ceh as a starting point introduces 


bias into tne procedure right from the sbart. second, the 


Meee 


extranovlation of growth factors from past developments 

a : q . 
is in itself a risky procedure’, and tiird, tue cast/ 
schedule relalionsnips are not sufficiently well defined 


to facilitate proper programming. 
Currently Based rxcursions 


Pne otyver eeneral "excursion" technique, rather 
than extrapolating from past developments, attempts to 
use Lie current siluation as tne basis for estimating 
cost growth. Typically, Delpni? tecnniques are used to 
accomplish this task. Yae Delpni technique uses a poll 
of qualified exnerts to arrive at a consensus estimate. 
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a basis for resource nrogrammirng. Pucy are, novwever, 
extremely valuable as a “cross check" and aro regularly 
used to validate that a SCH is at least a "“callpark" 
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"gazsed down" to the activity level. fnere is nothing in 
Hhe nrocedure to assist in getting good activity estimates. 
T™ fant, tne forn of tne required estimates (probability 
djcheitationws) represents an unusual anproacn co mosh 


Eee: BM SOUL ead tO Anadvertent: errors. 


na ‘Best Available Cop, 


” 


co ane meg. 


a enn ae WT, 


AlLuoure tna cnadel takes into gecount most seLivity 


ray 


jor 


2 


inLortvenendencies, il cannot wenerally uccount 
induced chances to follow on activities. for usaunle, 
if tue cost of ectivily 3 is dependent om tee caut of 
activily ", tne model will nonetheless treat tne two 


Betivity costs as indenendent. Special trealment by Lue 


analyst who develops the mode] may overcome some of 

these problems if the nature of tne dependency ic 

amenable to seecanodation by true model. 

Despite tno complexity of the model, jo still 

veoresentls a considerable simplifioalion of tre reul : 
a 

world. Mais stuaunmlifieation is aenieved tarougn various : 

assuuptions etther exvlicit aor implicit. in the model. 
i 

Mo tye dexroe thal tnece assunptions are not understood, : 


not recognized, or not accounted for in interpretation 
of resulls, tre mode] outnul may be nislowding and 
inaccurate, 

THere YS no Sinnle tecnnique for oerforming sensi- 
tivily analyses, xcursions form Lae vaseline inout 


require @ conrmlete rerun of tic Simulation. 
Overall @valuation 


Of all tre techniques currently in use, tne TRACE 
Fetwork lodeling Technique is by far the most compre- 


hensive, and offers tne most useful output, bovn in 


“7 


terms of HAC! application,flexibility, and ancillary 
uses of output (Ssucn as for project tsonitorsnip). line 
Network Modelins Veenanique furtner has the potential 
to improve to tne degree that network modeling and 
methods for developing better estimates of activity 
cost and schedules improve. ‘nus, where tne project 
cost justifies tne initial nign cost of developing tie 
TRACE Ketvork tiodel, and where appropriate analytic 
and computer support is available, the ketwork “iodel 


offers the best anprosen to VRACE development. 
TRACE RICK TA ULATTON - DeocaTPLloy 


wne Cockernan report takes note tnat in some cases 
the Network “Modeling Tecnnique may be inappropvriave. ‘ne 
TRACE Risk Tabulation technique is offered as an approacn 
to he used in such cases. It may be cnuaracterized as an 
expected value simulation of tne project, and like tne 
network approacn offers only an improved means of 
acerogvating a number of sub-estimates, witn no suggestion 
as to now tnoce critical sub-ostimates mignt be suarpened. 
Jhe technique is best explained by reference to tne tabu- 
lation (isure 11, which is tne example used in the 
Cockerian report). Step 1 involves relucins the project 
into elements suitable for analysia, ‘ine reporl sug 


that a conhination of Work breakdawn Structure (44.5) 
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elevents and critical milestones are Lest siiited for the 
Risk ebulakion Model. “he firct column in Pijgure 11. 
shows tho rpasults of sten ] for tne exanple provlen. 
Coliauans ? and 2? are completed by eecuring (hy belpoi 
techniques, orecumably) tue point prevability and 

east imracl of likely overruns to the program element 
lime. “hese are called ype A dwmpacts and may be inter 
preted as follows: (line 1) “Ynere is 4 0.2 probability 
tial the ernor ejement will experiance 4 cost averrun. 
If the overrun does oceur, the cost of tae armor will 
most Likely be inersasad py $3.9 million." 


Columns ',0,°6 provide for inaierdependencies (“ype 


jrpacts). For example, on line &, the colum 4 entry 


eee se Ce ee 


of .75 enns that af we do have probleme wita tae power 
traic, tnere is a .75 vronability that ovnuer prorram : 
elements will alto be affected, and if they are, tne 
overrur will ve </b,Q million with = two wonth se.edule 


Blin. (20146). ne ‘eolven £ e0lry. Venressacs. Uso overal T 


~ 


propailite thet thre eoluun 6 Jmpact will occur, amd is 


srk yay - obs A Ook , fae - maya oly “4 
eonmbed ac tre produst of the wype: A and ype 
ors 
‘, 


probabilities (.50 times .74). 


\. 


Column: 7? and ® indicate tne date and Fi.-cal Year 


of impact, to allow prover programminz of Place farcin 
Punds. Column 9 ic tus cxpected valuc,defined ws tue 


probability of impact times the nost of impact. Yor line 
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fe AO. CORRE LION 36) oh Pines hb Noe es AG aes 


Siowillion = 32.7 million. 


Goliers 10 GG can ad histuent. to Lhe expeclad value of 


impact. ‘ais adjustrent attempts to take into accounh ine 
Teoh a As oi ae ieee. OaGur, Po Ws Loan Lee il 
iinmact value - not the exveeted value. Phus af tre Vhs ve : 


VOPOY Were Cmte only lie expeeted yulite of tie twee, 
hf is Likely thab tie RACY “Marpir would be fiandequate. 


“he gontraccorp rrovides an hevristic for making tis 


Sas pule is tnat any impact with overall 
ercater Lian .75- will be Tully funded, ans 
trese with lesser prooadilities will use an adjusted 
expected valuc. For example, line 6, column 10 snows 


C2.2. million (g0.2 Trou esl. 2 yive £2.90 Trom col.G). 


eee. <1 as 


Sinee the probabilities ars greater than (or cqual to) 


0.75, full funding hes beer used. Line 4) provides an 
exanple of the adfuctrent proceedure wacn tus prohanil- 


Pa 
Sy 


» jeans trim .75. nere tee =lven probabilitics are 


oa 


Jase 
ct 
we 
co 
eo) 
» 
“3 


divided Vor end (as Cxpeoued. Velo PEcompulSed sis 
EG) p75) Bites ele Sie Cee. 75) BiaOs he: 25 sing. as 


ad jucved value 1s justified. or tie basis OF 2NCuRTivs 


felling anc “reaconableness". 
RACH STor PTASULAYWTOL - oD TO"! 
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Ac for tie Letvory fodeling porlion, ule discussed on 
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JS orpreniezed into a Jioting of aévintages, disadvantages 


and an overal] evaluation. 


Advantapres 


Moo techmigue is relatively inexnensive and 


readily ceasy te ayplo. 
\ 
Updating ic roadily accomplished, 


Sensitivity analysis 1s particularly simple since 
ine conbrimatlion of any clement to tun wean can be 
aualvtically computed. Vor example, Suppose we wisn ts 
exanine tao impact of enansing the probability estimute 
for element 1 from .2 toa 4. Vae new expected logs is 
Sho ein: Cel eee So a i, eas tae ae 


eznectoad: Joss is Fi.f mildion (.''/.75 times (2 million). 


Disadvantarces 


Vane teenmique uses point probanilities of point 
9035 as Sub-estimates. This 15 @ sross simplification 
of the real world and can lead to large errors. 

wae technique is hignly dependent on the skill of 


\- 
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ween ee eee ere 


tric analyet to identify ard account! for tse various 
interdependoncies, 

The oitpul iso @ point ectimale wiich is always 
sreaber toa Tie wean, Put which eanniou be retailed to 
the desired value of tie TRACE (the incdian of tiie 
distribution), 

secauce the leyel of apzregation is high, the 


veonunique is quite sensitive Lo errors in the sun- 


Cyerall tUvaluerion 


in comparison witht tne Network Nodeling: Approse.:, 


the Risk Tabulation metnod apyears weak, with tne dis- 


advantaves far oubweishine the advantaszes. However, in 
comparicon to the previous teenniques, it dccs provide 
a well defined, structuren. teclmigue wiles forces<= 
dezrec of diseiplined analysis into tue problem. For 


tah reason it is preferred to earlier tecuuiigques. 


fowever, with ninor modifica:ion, tie S8iCu 215k Vabula- 


Lion method cun be improved. A proposal for cue. dinprave- 


mont is vresentea in tne next chapter. 
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two of the major disadvantages of the Risk Tabula- 
tion metnod are the use of point estinates us output and 
tee “ouristiec adjusbment of the expected values to pet 
vee final ovtout. A modified Leannique is proposed whicu 


overcoies Toth tnesc disadvantases. 


The initial foraulation of the nroclem (figure 1?) 


2 


is tee same au for tie TRAC! Risk Tabulation. siovever,. 
tris conpatacion of tne expected value and adjustment 
tneroto is deleted, Tnutead, avery Single cmanuler pro- 
epan (Annex °) 18 used Lo axuand the program element 
estimates into teeter naturel wrobability districulion, 


which 45 cisnly an oclLimate of tne probublility disvtri- 


pmabion of overall project cout. 


Uri the individual prone ni livies “vem in Mesure 
1?, tee protasjility distribulion 13 eomputed in a 
straje ¢forvard manner: 

All poositle combinations of impscus and non.. 


Langeto are enumerated and the assoclabled propabilities. 
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and costs are commuted .As an example, one of tie 78 


i ad A As eM agi al 


possitle eambinations in tne example proplem (fipure 12) 
is: no impact in e@elenents 1,2,3,5,6,and @; a type A impuct : 


in elerent 4; and tyoe A and 3 impacts in elements 7 and 


9. The promability of this combination would ve computed 


side Ue.) C12 16) 15525) COIS e275 G7 EO eI CS) 


Hm ek SO 


witen is .000605. The associated cost is 311.5 million, 


consistine of 21.5 fron element 4 (type A), <6 from 
elomernt 7 {type A *% 3) and ,' from clement 9 (type a « 3). i 

\fter all possible compinations are evalualed, trey 
are arranzed in aseendinzs order of cost, those combin- 
ations with equal costs aro combined oy eddins tegetner : 
tnieir pronabilitiec, and a cumulative total of probahbili- - 
tias in also commuted. ievres 73 and 74 display tue 
results of tne process for tne asmaple proslem. Pi 
13 dicplays the innul data for validation along wita the. 
varjanen and standard deviation of tric distribution, 


"ean, 


(these quantitiec are computed by the provramn as u side 
benefit), wiile figure 14 is the listins of tne distri- 
bution per go, 

Tre conpyter program also produces two other output 
rormats for cpeeisl purposes. ne first is a very detailed 
breakout Snowing each of ths noseible conbinations of 
jmnacts alons with tne overall] impact contribution 


=“ 


(Annex ©). nis output is designed for the analyst who 
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DEPARTMENT OF THE ARMY 
OFFICE OF THE ASSISTANT SECRETARY 
WASHINGTON, D.C. 20310 


12 JUL 1974 


MEMORANDUM FOR THE DIRECTOR OF THE ARMY STAFF 


SUBJECT: RDT&E Cost Realism -- Future Development Programs 


One of the strongest criticisms leveled against the Department of 
Defense in recent years has, I believe, related to the difficulty of 
controlling the costs of development activities. The present memo- 
randum addresses development costs; however, the action proposed 
could equally apply to the estimation of production "design-to" costs. 


It is noted that the problem in the past has not been the need of a 
method for fine-tuning cost estimates to, say, the nearest 2 or 3 or 
even 10%. Rather, the problem has been one of finding a means of 
avoiding gross errors on the order of 50 to 100% and more. We have 
rarely, if ever, had an underrun ona major program, However, it 
is not our objective to establish program cost estimates so high that 
tnere would be underruns in a large majority of cases. Our estimate 
should. be unbiased so that we have about an even chance of either 
going over or under it. —— 

De 
It is submitted that cost overruns will continue to be a way of life 
until adequate recognition is given to the impact of program uncertain- 
ty in estimating costs, There are, of course, many types of uncer - 
tainties that influence program costs, such as unforeseen program 
changes, inflation, inaccurate cost estimating, technical problems, 
threat revisions, etc, It is particularly important for the Army to 
improve our capability as a "sophisticated buyer" in the analysis, 
evaluation and validation of program proposals by industry. 


RDT&E cost estimates have traditionally been based on an approach 
which compiles expected costs for specifically identified tasks. It is, 

of course, important that we continue co refine our ability to identify 
those essertial tasks and to prepare plars for their efficient accomplish- 
ment. Buy beyond this, it remains the fundamental nature of RDT&E, 
wherein by definition one is producing a particular item for the first 
time, to involve the unknown. These unknowns invariably lead to cost 
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requirements which cannot be individually foreseen at the outset of 

a development -- yet their cumulative impact can be seen in retrospect 
with all the assuredness of the lav’s of probability, The record shows 
a major factor in properly estimiting RDT&E cost is, therefore, to 
assess realistically the probabl- gross extent of the unknowns in a 
‘program in advance -- i,e., the certain uncertainties, 


The lessons of history strongly suggest that even the most capable 
program manager is not able to forecast all the problems which will be 
-encountered in a development program wh.cth typically spans six to ten 
years into the future, It is, however, quice possible to conduct a risk 
analysis and thereby forecast, statistically, the probable additional 
funds which will be required to overcome these certain uncertainties. 


The provision of flexibility in the funding plan for such purposes does 


not represent the creation of a ''slush fund," or 2 Ureserve" (in the 


usual context), but t rather merely gives due rec ognition to to the fact that 
, unanticipated problems always have, and aiways will, arise even in the 


‘best managed RDT&E programs... and that our base-line cost esti- 


“mates should reflect these probable additional costs. 


The provision of additional planning funds commensurate with the 
‘assessed risk in proposed programs does, of course, mean that the 
effort to justify these new programs will be more dificult. Similarly, 
it will not be possjble to start as many new programs if they must all 

fit into a given out-year budget projection, On the other hand, it seems 
far preferable to recognize these realities at the outset rather than part 
way through a development at which time cost growth in one program 
can often be compensated only by the inefficient mechanism of cancelling 
still other partly completed programs. 


It is, therefore, proposed that aTotal Risk Assessing Cost Estimate 
(TRACE) be generated for all future development programs and used as 
the basis for justifying those programs. The TRACE estimate is defined 
as one having a 50:50 chance of producing either an overrun or an under- 
run. One possible method of aggregating the TRACE estimate is: 


(1) A "risk factor’' would be assigned to each element of the 
program's work breckdown structure (probably a three-level structure 


is adequate for this purpose and should be available early in the program). 
A risk factor of 1.0 would correspond to a routine task with no risk 
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whatsoever, A fact:..r of 1,3, for example, would characterize an effort 
wherein experience ‘ndicates that tasks which are not explicitly definable 
at the outset will m.sst probably exceed those which are, in fact, 

definable by an exp: -cted value of about 30%. (The actual magnitude of 
the risk factor can :;enerally be determined by a comparison of the degree 
of departure of the proposed task from corresponding tasks which have - 
been accomplished in the past and by an assessment of the uncertainty - 

_ which exists in the performance demanded of the item.) 


; (2) The risk factor would be multiplied by the basic cost esti- 
mate for the known efforts to be performed in each element of the work 
’ breakdown structure. An aggregate of these weighted costs would then 
provide a cost estimate for the program which recognizes the certain 
uncertainties which invariably arise in a development program. 


(3) The TRACE estimate would make use of all supportable 
estimating techniques such as Independent Parametric Cost Esti- 
mating (IPCE), 


(4) The forecast of fund requirements which has been defined 
after consideration >f uncertainties ir. the above manner would 
generally (although +t necessarily always) be included in the latter 
years of the develop. -2nt funding plan. By so doing it is not necessary 
to justify the budget ; of these funds to Congress in a given year until 
the specjfic reason '.r needing them has, in fact, become known. 
Further, if, :v the .. v of averages, some of the funds on an individual 
program are ot ult\.aately needed, they can simply be dropped from the 
then-year bu. “yet re quest before it is submitted to Congress. 


(5) Th: development activity would continue to be managed on 
a day-to-day i.asis against the nominal cost estimate (i,e., without 
risk funding), making the additional funds available to the Project 
Manager only vhen formally requested and adequately justified. 

& 

(6) Al] new R&D programs will be discussed with the user and 
justified to OSD and to the Congress using the TRACE figure (i.e., the cost 
estimate including the expected impact of risk), Program threshold cost 
estimates in t! 2 DCP would be no less than the TRACE estimate and 
generally grealer, 


kn 
xo 


This concept is shown schematically in the following sketch: 
DCP threshold | 4 
is in doubt, 
Trace Cost ? 
(to be used in FYDP anc 
program justification) 


effort, 


Nominal Cost 
(formerly used in FYDP 
and program justifica- 
tion, Still used as basic 
target in day-to-day 
program management. ) 


tasks. 


In summary, it snould be emphasized that the approach proposed herein 


« is not one of creating unverified contingency funds or slush funds or self- 


fulfilling cost estimates, Tothe contrary, it merely gives recognition to 


: the established fact that every individual funding demand which will arise 


n 2 6-10 year development effort simply cannot be explicitly identified 
in advance -- but that the aggrcgate of those demands can be reasonably 
predicted and should, therefore, be included in program planning. 


~ It should be noted that the problem being addressed herein is related to, 


but intentionally s#parated from, .the also very difficult problem of pre- 
cicting inflation. This latter task need not be allowed to complicate the 
matter considered herein of properly accounting for those factors which 
are largely within the control of program management. 


Unless you have some objection to the policy and procedures expressed 
above, please have them implemented in appropriate Army materiel 
acquisition directives. 


: re. 
‘Norman R. & Chg Tar 


Assistant Secretary of the Army 
(Research and Development) 
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) Cost margin before program continuation 


1) Cost of statistically probable additional 


Z )Cost of clearly identifiable and verifiable 
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SECRETARY OF THE ARMY 


WASHINGTON Ms 


July 12, 1974 


MEMORANDUM FOR THE CHIEF OF STAFF 


SUBJECT: RDT&E Cost Realism -- Current and Future 
Development Programs 


Past experience in-the acquisition of major weapons “systems 
indicates that we generally exceed our initial cost estimates ... 
sometimes by as much as 100% or more. This problem, of course, 


--is not unique to the Army. However, to prevent misleading the 


decision makers who approve our programs, we must produce more 


_ accurate cost estimates for the development of our major weapons 


systems. 


The accuracy of our major program cost estimates is 
influenced by many factors including inflation, poor estimating, 
changing requirements, technical risk, etc. Risk is inherent in any 
research and development program and, generally, the greater the 
risk, the greater the potential payoff of the program.  Unfortunaiely, 
there also appears to be a direct correlation hetween the degree of 
risk and the difficulty of producing accurate cost estimates. 

While we gannot eliminate all technical risk from our pro- 
grams nor precisely predict specific problems that will occur, we 
can, from past experience, assess their impact statistically and 
consider this in our RDT&E program plans. The problem of 
generating RDT&E cost estimates that mire accurately reflect 
probable outcomes has ur ergone a review by elements of the 
Secretariat and the Army Staff and a revised policy has resulted. 
Under this revised policy we would no longer follow the practice 
of programming based only upon those tasks which can be clearly 
foreseen at the outset of a program and which therefore do not 
allow for uncertainty. By applying this Total Risk Assessing Cost 
Estimate (TRACE) approach, I am convinced that we can provide 
more accurate data in support of our programs at the OSD and 
Congressional levels. I have asked tre ASA(R&D) to provide the 
details of the TRACE concept directly to your staff and would be 
appreciative of your helping implement this policy guidance. 
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DEPARTMENT OF THE ARMY 


OFFICE OF THE CHIEF OF STAFF 
WASHINGTON, D.C. 20310 


5 August 1974 


MEMORANDUM FOR: SECRETARY OF THE ARMY 


SUBJECT: RDTE Cost Realism -- Current and Future Development Progra:.:: 


o 


We have begun the process of implementing the policy of RDTE cost realism 
enunciated by you and Assistant Secretary Augustine in your 12 July 1974 
memoranda. Your policy will receive high visibility here and in the 
field. 


For each current development program qualifying for decision by the 
ASARC, the Chief of Research, Development and Acquisition is requesting 
@ one-page review from the materiel developer. Each review will show 
what changes would be needed in a program to provide 10% and 20% margins 
of estimated funding from FY76 to completion of development. These 
reviews will be evalua:ed at HQDA by 30 September 1974 for the purpose 
ef making specific pro, .am recommendations. 


For future development | -ograms, the Chief oi Research, Development and 
Acquisition, working wii i tne Comptroller of the © y (Director of Cost 
Analysis), will issue .' ‘etter of instruction on 1 October 1974. This 
wiJl be an inter:n doc. nt to tmplesent your policy, pending revision 
of Army regulati:is. 


A 3; 
5 ie 
FRED C. WEYAN 
‘ General, UnjXed States Army 
: Vice Chief of Staff 
6 
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